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Abstract 


This  paper  reports  on  the  first  application  of  Hierarchical  Goal  Analysis  (HGA)  [1],  a  relatively 
new  approach  to  requirements  analysis  for  complex  systems,  to  naval  command  and  control. 
HGA,  applied  to  1 1  positions  of  the  Canadian  Forces  Halifax  Class  Frigate  operations  room, 
decomposed  three  top-level  goals  to  a  full  goal  hierarchy  of  563  goals.  The  hierarchy  ranged  from 
four  to  nine  levels  deep,  with  an  operator  assigned  to  each  goal.  The  HGA  process  concluded 
with  a  stability  analysis  for  identifying  potential  goal  conflicts  and  an  upward  flow  analysis  for 
identifying  requirements  for  feedback  between  operators.  An  examination  of  the  stability  analysis 
revealed  that  the  current  design  of  the  operations  room  includes  few  sources  of  instability  where 
multiple  operators  compete  for  control  of  the  same  variable.  The  upward  flow  analysis  revealed 
that  the  requirement  for  feedback  from  operators  assigned  to  lower-level  goals  to  operators 
assigned  to  higher-level  goals  is  relatively  high,  and  the  operations  room  could  benefit  from 
review  and  redesign.  The  goal  hierarchy,  operator  assignments,  stability  and  upward  flow 
analyses,  and  proposed  solutions  were  reviewed  by  subject  matter  experts.  While  used  to  model 
an  existing  system,  the  present  application  of  HGA  appears  to  be  especially  useful  in  providing  a 
basis  for  evaluating  a  system  design  and  developing  design  recommendations. 


Resume 


Le  present  document  decrit  la  premiere  application  de  F  analyse  des  objectifs  hierarchiques 
(AOH)  [1],  une  methode  relativement  nouvelle  d’analyse  des  besoins  pour  les  systemes 
complexes,  au  commandement  et  au  controle  navals.  L’AOH,  appliquee  all  postes  de  la  salle 
des  operations  de  la  fregate  de  classe  Halifax  des  Forces  canadiennes,  a  decompose  trois  grands 
objectifs  en  hierarchie  complete  de  563  objectifs  qui  comportaient  quatre  a  neuf  niveaux,  un 
operateur  etant  associe  a  chaque  objectif.  L’AOH  s’est  conclue  par  une  analyse  de  stabilite  qui 
permet  de  cemer  les  conflits  potentiels  d’objectifs  et  une  analyse  ascendante  qui  permet  de 
determiner  les  besoins  de  retroaction  entre  les  operateurs.  Un  examen  de  l’analyse  de  stabilite 
indique  que  l’amenagement  actuel  de  la  salle  des  operations  compte  peu  de  sources  d’instabilite 
la  ou  de  nombreux  operateurs  se  font  concurrence  pour  le  controle  de  la  meme  variable. 
L’analyse  ascendante  a  demontre  que  le  besoin  de  retroaction  pour  les  operateurs  responsables 
des  objectifs  des  niveaux  inferieurs  et  les  operateurs  responsables  d’objectifs  d’un  niveau  plus 
eleve  est  relativement  grand,  et  que  la  salle  des  operations  pourrait  beneficier  d’un  examen  et 
d’une  restructuration.  La  hierarchie  des  objectifs,  les  taches  des  operateurs,  les  analyses  de 
stabilite  et  ascendante  et  les  solutions  proposees  ont  ete  examinees  par  des  experts  en  la  matiere. 
Meme  si  elle  a  servi  a  modeliser  un  systeme  en  place,  F  application  actuelle  de  l’AOH  semble  etre 
un  point  de  depart  particulierement  utile  pour  evaluer  la  conception  de  systemes  et  formuler  des 
recommandations  relatives  a  la  conception. 
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Executive  summary 


Application  of  Hierarchical  Goal  Analysis  to  the  Halifax  Class 
Frigate  operations  room 

Renee  Chow;  Jacquelyn  Crebolder;  DRDC  Toronto  TR  2007-161;  Defence  R&D 

Canada  -  Toronto;  November  2007. 

Introduction  or  background:  This  paper  reports  on  the  first  application  of  Hierarchical  Goal 
Analysis  (HGA)  [1]  to  naval  command  and  control.  HGA  is  a  relatively  new  approach  to 
requirements  analysis  for  complex  sociotechnical  systems.  It  uses  goals  (i.e.,  desired  system 
states)  rather  than  tasks  (i.e.,  operator  activities)  or  functions  (i.e.,  engineered  purposes)  as  the 
primary  unit  of  analysis.  Goals  are  identified  through  a  process  of  hierarchical  decomposition.  For 
each  goal,  HGA  identifies  a  controlled  variable  whose  state  is  compared  to  the  goal  and 
manipulated  as  required  to  meet  the  goal.  Once  all  goals  and  their  corresponding  controlled 
variables  have  been  identified,  each  goal  is  assigned  to  a  human  or  automated  operator. 

In  the  current  application,  HGA  was  applied  to  a  Canadian  Forces  Halifax  Class  Frigate 
operations  room  to  inform  on  the  modernization  of  its  workspace  design  and  its  command  and 
control  systems.  HGA  was  chosen  as  an  appropriate  method  of  analysis  for  this  particular 
application  because  of  its  potential  to  consider  and  integrate  new  and  changing  aspects  of  the 
environment,  such  as  new  positions  in  the  operations  room  and  re-allocation  of  roles  between 
human  operators  and/or  automated  systems. 

Results:  Eleven  positions  of  the  current  Halifax  Class  Frigate  operations  room  team  were 
analyzed  using  HGA.  Three  top-  level  goals  were  decomposed  to  a  full  goal  hierarchy  of 
563  goals  ranging  from  four  to  nine  levels  deep.  The  analysis  revealed  substantial  variation  in  the 
number  of  goals  assigned  to  each  operator.  Of  the  1 1  operators  analyzed,  the  Operations  Room 
Officer  (ORO),  the  Sensor  Weapon  Controller  (SWC),  and  the  Assistant  Sensor  Weapon 
Controller  (ASWC)  accounted  for  nearly  half  the  goals.  In  contrast,  the  Commanding  Officer 
(CO)  was  only  assigned  to  34  goals,  but  these  included  all  seven  first-level  goals.  Due  to  the 
HGA  process,  additional  operators,  such  as  the  Shipbome  Aircraft  Controller  (SAC)  and  the 
Officer  of  the  Watch  (OOW),  were  recognized  as  positions  that  should  be  included  in  the  analysis 
if  an  understanding  of  the  overall  system  is  desired. 

The  HGA  included  a  stability  analysis  which  identified  seventeen  sources  of  potential  instability 
in  the  operations  room,  where  the  same  variable  was  associated  with  multiple  goals  assigned  to 
multiple  operators.  Recommendations  to  resolve  each  source  of  instability  were  proposed, 
ranging  from  changes  to  the  workspace  layout,  to  interface  design,  to  the  implementation  of  new 
supporting  technologies  or  information  exchange  procedures.  The  HGA  also  included  an  upward 
flow  analysis  which  identified  776  requirements  for  feedback  from  one  operator  (assigned  to  a 
supporting  goal)  to  another  operator  (assigned  to  the  supported  goal).  These  requirements  were 
rated  by  domain  experts  based  on  importance,  frequency,  and  the  modality  (i.e.,  visual  or  verbal) 
in  which  they  were  being  addressed.  Considering  only  the  primary  operators  for  each  goal,  85% 
of  feedback  requirements  were  found  to  be  addressed  verbally,  pointing  to  potential  overloading 
of  the  auditory  channel,  and  the  possibility  for  new  visual  aids  to  provide  operators  with  the 
necessary  feedback.  The  highest  numbers  of  overall,  important,  and  frequent  requirements  for 
feedback  were  found  between  the  SWC  and  the  ORO,  substantiating  the  need  for  these  two 


DRDC  Toronto  TR  2007-161 


operators  to  be  co-located,  and  thereby  providing  a  basis  for  prioritizing  the  co-location  of  other 
operators. 

Significance:  Although  HGA  was  conceived  and  first  applied  in  the  aviation  world,  this  study 
demonstrates  the  applicability  of  the  approach  in  a  naval  context.  It  also  demonstrates  that  HGA 
could  be  used  to  analyze  a  system  consisting  of  a  large  number  of  operators,  thereby  establishing 
it  as  a  suitable  methodology  for  the  analysis  of  other  complex  military  systems,  such  as  the  multi- 
organizational  structure  of  maritime  security.  Despite  the  size  and  complexity  of  the  operations 
room,  HGA  was  able  to  identify  potential  goal  conflicts  as  well  as  the  needs  for  information  flow 
between  operators.  HGA  provided  a  systematic  method  for  describing  the  dependencies  between 
operators  in  both  qualitative  terms  (e.g.,  the  hierarchical  relations  between  goals,  the  attributes  of 
each  goal,  etc.)  and  quantitative  terms  (e.g.,  the  number  of  operators  who  may  compete  for 
control  of  a  specific  variable,  the  number  of  feedback  requirements  between  two  specific 
operators,  etc.).  It  was  then  possible  to  describe  the  design  of  the  existing  operations  room  in 
terms  of  its  level  of  support  for  these  various  dependencies.  The  authors  anticipate  that  the  same 
method  can  be  applied  to  describe  alternate  designs  of  the  operations  room,  so  that  they  can  be 
compared  with  one  another  or  with  the  existing  design.  Different  criteria  (e.g.,  support  for  the 
provision  of  important  feedback  versus  frequent  feedback)  can  be  applied  to  compare  designs. 

Future  plans:  The  knowledge  elicitation  phase  of  this  HGA  was  challenging,  as  subject  matter 
experts  (SMEs)  were  accustomed  to  thinking  in  terms  of  tasks  rather  than  goals.  The  development 
of  the  goal  hierarchy  and  the  documentation  of  goal  attributes  were  conducted  using  a  tool 
designed  for  hierarchical  task  analysis,  and  the  tool  did  not  include  support  for  translating  the  data 
collected  on  goals  directly  into  stability  analysis  and  upward  flow  analysis  outputs.  To  support 
future  applications  of  HGA,  the  development  of  training  modules  on  HGA  and  a  software  tool  for 
performing  all  phases  of  HGA  would  reduce  the  time  and  effort  required,  improve  the  usability  of 
the  outputs,  and  improve  the  re-usability  of  previous  related  analyses. 

The  current  application,  as  well  as  a  previous  application  of  HGA  to  the  control  of  unmanned 
aerial  vehicles,  analyzed  cognitive  work  at  the  tactical  level.  Future  work  should  explore  the 
applicability  of  HGA  to  analyzing  cognitive  work  at  the  operational  or  strategic  levels. 
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Sommaire 


Application  de  I’analyse  des  objectifs  hierarchiques  a  la  salle 
des  operations  de  la  fregate  de  classe  Halifax  :  line  etude  de  cas 


Renee  Chow;  Jacquelyn  Crebolder;  DRDC  Toronto  TR  2007-161;  R  &  D  pour  la 
defense  Canada  -  Toronto;  Novembre  2007. 

Introduction  ou  contexte  :  Le  present  document  decrit  la  premiere  application  de  1’ analyse  des 
objectifs  hierarchiques  (AOH)  [1]  au  commandement  et  au  controle  navals.  L’AOH  est  une 
methode  relativement  nouvelle  d’ analyse  des  besoins  pour  les  systemes  sociotechniques 
complexes.  Elle  utilise  des  objectifs  (c.-a-d.  l’etat  desire  de  systeme)  plutot  que  des  taches  (c.-a-d. 
les  activites  des  operateurs)  ou  des  fonctions  (c.-a-d.  objectifs  techniques)  comme  unites 
d’analyse  de  base.  Les  objectifs  sont  definis  par  decomposition  hierarchique.  A  chaque  objectif 
correspond  une  variable  controlee  qui  lui  est  comparee  et  qui  est  manipulee  de  fag  on  que 
l’objectif  soit  atteint.  Apres  avoir  defini  tous  les  objectifs  et  les  variables  controlees  qui  y 
correspondent,  on  attribue  chaque  objectif  a  un  operateur  humain  ou  automatise. 

Dans  le  cas  present,  l’AOH  a  ete  appliquee  a  la  salle  des  operations  d’une  fregate  de  classe 
Halifax  des  Forces  canadiennes  afm  de  guider  la  modernisation  de  l’amenagement  de  l’espace  de 
travail  et  des  systemes  de  commandement  et  de  controle.  L’AOH  a  ete  choisie  comme  methode 
appropriee  d’analyse  pour  cette  application  particuliere  en  raison  de  son  potentiel  de  tenir  compte 
et  d’integrer  des  nouveaux  aspects  changeants  de  l’environnement,  telles  que  des  nouveaux  postes 
dans  la  salle  des  operations  et  la  reaffectation  des  roles  entre  les  operateurs  humains  et/ou  les 
systemes  automatises. 

Resultats  :  Onze  postes  de  l’equipe  de  la  salle  des  operations  actuelle  d’une  fregate  de  la  classe 
Halifax  ont  ete  soumis  a  l’AOH.  Trois  grands  objectifs  ont  ete  decomposes  en  hierarchie 
complete  de  563  objectifs  qui  comportaient  quatre  a  neuf  niveaux.  L’analyse  a  revele  que  le 
nombre  d’objectifs  attribues  a  chacun  des  operateurs  variait  considerablement.  Parmi  les  onze 
operateurs  analyses,  l’officier  de  la  salle  des  operations  (OSO),  le  controleur  d’armes  par  capteurs 
(CAC)  et  1’ assistant  du  controleur  d’armes  par  capteurs  (ACAC)  ont  compte  pour  presque  la 
moitie  des  objectifs.  Par  comparaison,  le  commandant  ne  s’est  vu  attribuer  que  34  objectifs,  qui 
comprenaient  toutefois  les  sept  objectifs  du  premier  niveau.  Vu  le  processus  de  l’AOH,  il  a  ete 
etabli  que,  pour  comprendre  le  systeme  dans  l’ensemble,  il  faudrait  ajouter  a  l’analyse  d’autres 
operateurs  comme  le  controleur  d’aeronefs  embarques  et  l’officier  de  quart. 

L’AOH  a  compris  une  analyse  de  stabilite  qui  a  permis  de  reconnaitre  dix-sept  sources 
d’instabilite  potentielle  dans  la  salle  des  operations,  ou  la  meme  variable  est  associee  a  de 
multiples  objectifs  attribues  a  des  operateurs  differents.  On  a  fait  des  recommandations  pour 
corriger  chaque  source  d’instabilite,  allant  de  changements  a  l’amenagement  de  l’espace  de 
travail  et  de  la  conception  des  interfaces  a  la  mise  en  place  de  technologies  de  soutien  ou  de 
procedures  d’echange  de  renseignements.  L’AOH  a  egalement  compris  une  analyse  ascendante 
qui  a  permis  de  cerner  776  besoins  de  retroaction  de  la  part  d’un  operateur  (affecte  a  un  objectif 
de  soutien)  a  un  autre  operateur  (affecte  a  un  objectif  de  soutien).  Ces  besoins  ont  ete  evalues  par 
des  experts  du  domaine  selon  l’importance,  la  frequence  et  le  moyen  (c.-a-d.  visuel  ou  verbal)  par 
lequel  on  les  satisfait.  En  limitant  l’analyse  aux  operateurs  principaux  de  chaque  objectif, 
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85  p.  100  des  retroactions  se  font  verbalement,  ce  qui  pourrait  faire  craindre  une  saturation  dans 
le  conduit  auditif  et  laisser  entrevoir  l’utilisation  possible  de  nouvelles  aides  visuelles  pour  fournir 
aux  operateurs  la  retroaction  necessaire.  C’est  entre  le  CAC  et  l’OSO  qu’ont  ete  observees  les 
retroactions  les  plus  importantes  et  les  plus  frequentes;  ce  qui  confirme  la  necessite  pour  ces  deux 
operateurs  de  travailler  au  meme  endroit  et  justifie  le  fait  d’accorder  la  priorite  au  regroupement 
d’autres  operateurs. 

Importance  :  Meme  si  l’AOH  a  ete  conpue  et  appliquee  d’abord  en  aviation,  la  presente  etude 
demontre  que  la  methode  peut  etre  appliquee  dans  un  contexte  naval.  Elle  demontre  egalement 
que  l’AOH  peut  servir  a  analyser  un  systeme  qui  se  compose  d’un  grand  nombre  d’operateurs, 
faisant  ainsi  de  cette  analyse  une  methode  appropriee  pour  analyser  d’autres  systemes  militaires 
complexes,  tels  que  la  structure  multi-organisationnelle  de  la  securite  maritime.  Malgre  la  taille  et 
la  complexite  de  la  salle  des  operations,  l’AOH  a  permis  de  cemer  les  conflits  potentiels 
d’objectifs  ainsi  que  les  besoins  en  matiere  de  cheminement  de  l’information  entre  les  operateurs. 
L’AOH  a  foumi  une  methode  systematique  pour  decrire  la  dependance  entre  les  operateurs  en 
termes  qualitatifs  (p.  ex.  les  liens  hierarchiques  entre  les  objectifs,  les  caracteristiques  de  chaque 
objectif,  etc.)  et  en  termes  quantitatifs  (p.  ex.  le  nombre  d’operateurs  qui  peuvent  se  faire 
concurrence  pour  le  controle  d’une  variable  precise,  le  nombre  de  besoins  de  retroaction  entre 
deux  operateurs  precis,  etc.).  II  a  ensuite  ete  possible  de  decrire  l’amenagement  de  la  salle  des 
operations  actuelle  en  fonction  de  son  niveau  de  soutien  pour  ces  diverses  dependances.  Les 
auteurs  prevoient  que  la  meme  methode  peut  etre  utilisee  pour  decrire  d’autres  amenagements  de 
la  salle  des  operations,  pour  que  Ton  puisse  les  comparer  entre  eux  ou  avec  l’amenagement 
actuel.  On  peut  appliquer  differents  criteres  (p.  ex.  soutien  pour  la  retraction  importante  par 
opposition  a  la  retroaction  frequente)  pour  comparer  les  amenagements. 

Plans  futurs  :  La  phase  d’elicitation  de  la  connaissance  de  l’AOH  s’est  averee  difficile,  puisque 
les  experts  en  la  matiere  etaient  habitues  a  penser  en  termes  de  taches  plutot  que  d’objectifs. 
L’ elaboration  de  la  hierarchie  des  objectifs  et  la  description  des  caracteristiques  des  objectifs  ont 
ete  effectuees  a  l’aide  d’un  outil  congu  pour  l’analyse  des  taches  hierarchiques,  et  l’outil  ne 
comprenait  pas  de  soutien  pour  convertir  directement  les  donnees  recueillies  sur  les  objectifs  en 
resultats  d’analyse  de  stabilite  et  d’analyse  ascendante.  Afm  d’appuyer  l’application  future  de 
l’AOH,  le  developpement  de  modules  de  formation  sur  l’AOH  et  un  outil  logiciel  pour  effectuer 
toutes  les  phases  de  l’AOH  reduirait  le  temps  et  1’ effort  requis,  ameliorerait  la  convivialite  des 
resultats  ainsi  que  la  reutilisation  des  analyses  connexes  precedentes. 

L’application  actuelle,  ainsi  que  l’application  precedente  de  l’AOH  au  controle  de  vehicules 
aeriens  sans  pilotes,  a  analyse  le  travail  cognitif  au  niveau  tactique.  Les  travaux  futurs 
doivent  porter  sur  l’applicabilite  de  l’AOH  a  1’ analyse  du  travail  cognitif  aux  niveaux 
operationnel  ou  strategique. 
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1  Introduction 


1.1  Background 

In  the  human  factors  literature,  different  approaches  have  emerged  for  the  analysis  of  complex 
cognitive  systems  to  identify  requirements  for  the  design  of  workspaces,  displays  and  controls, 
decision  support  systems,  and/or  computer  supported  cooperative  work.  Many  of  these 
approaches  have  used  either  tasks,  defined  as  activities  performed  by  human  operators  (cfi,  [2], 
[3],  [4])  or  functions,  defined  as  capabilities  afforded  by  a  system  and  its  components  (cfi,  [5], 
[6],  [7])  as  the  primary  units  of  analysis.  Recently,  Hendy,  Beevis,  Lichacz,  and  Edwards 
proposed  a  contrasting  approach  called  Hierarchical  Goal  Analysis  (HGA)  [1].  Rather  than  tasks 
and  functions,  HGA  uses  goals,  defined  as  desired  states  for  variables  that  are  monitored  and 
controlled  by  human  operators  or  automation  in  a  system,  as  the  primary  units  of  analysis.  This 
paper  reports  on  the  first  application  of  HGA  to  naval  command  and  control,  specifically  the 
analysis  of  1 1  operator  positions  in  the  operations  room  of  the  Halifax  Class  Frigate.  There  had 
been  a  previous,  less  complex  application  of  HGA  to  analyze  the  control  of  Uninhabited  Aerial 
Vehicles  (UAVs)  [8,  9]. 

The  current  analysis  served  two  purposes.  On  one  hand,  it  served  as  a  case  study  for  investigating 
the  applicability  of  a  relatively  new  analytical  approach  to  a  problem  domain  where  it  had  never 
been  applied.  The  lessons  learned  provide  some  information  for  other  human  factors  analysts  to 
consider,  as  they  choose  between  HGA  and  other  approaches  for  analyzing  their  systems  of 
interest.  On  another  hand,  this  analysis  identified  needs  and  opportunities  for  changes  to 
the  current  operations  room  of  the  Halifax  Class  Frigate,  providing  some  information  for 
the  Canadian  Navy  to  consider  as  they  look  towards  developing  their  capabilities  in  support  of 
future  operations. 

1.2  Contrasting  approaches 

While  HGA  is  not  the  first  approach  to  analyze  goals,  there  are  some  key  contrasts  between  HGA 
and  previous,  more  well-known  approaches  such  as  Hierarchical  Task  Analysis  (HTA)  [2]  and 
Goal-Directed  Task  Analysis  (GDTA)  [10].  In  general,  HTAs  or  GDTAs  are  performed  for  a 
specific  operator,  class  of  operators,  or  team  of  operators,  so  the  identification  and  decomposition 
of  goals  are  based  on  the  roles  and  responsibilities  that  have  been  assigned  to  the  operator(s).  If 
an  operator’s  role  is  modified,  goals  may  need  to  be  added  and/or  removed  from  the  hierarchy.  In 
contrast,  an  HGA  is  performed  for  a  system  with  an  unspecified  number  of  operators  belonging  to 
different  classes  and/or  teams.  It  identifies  and  decomposes  all  goals  before  any  goal  is  assigned 
to  any  operator.  Even  if  there  are  changes  to  the  role(s)  assigned  to  the  operator(s),  the  goal 
hierarchy  itself  does  not  need  to  be  revised.  So  while  HTA  and  GDTA  readily  support  design  in 
the  context  of  mature,  existing  systems  and  organizations  (i.e.,  ones  where  goals  are  assigned  to 
roles),  the  process  followed  in  an  HGA  seems  to  offer  the  flexibility  required  to  support  the 
design  of  “envisioned  worlds”  (cfi,  [11],  [12]),  where  the  number,  types,  and  roles  of  operators 
may  be  undecided  or  subject  to  change. 

This  flexibility  offered  by  HGA  may  also  come  at  an  analytical  cost.  HGA’s  emphasis  on  goals 
and  variables  may  complicate  the  knowledge  elicitation  phase  of  the  analysis.  Subject  matter 
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experts  (SMEs)  may  have  little  difficulty  in  answering  questions  about  what  tasks  they  perform  or 
what  decisions  they  make  (as  in  an  HTA  or  a  GDTA,  respectively)  to  meet  goals,  but  they  may 
find  questions  about  what  variables  need  to  be  monitored  and  what  states  these  variables  need  to 
achieve  (which  must  be  asked  in  an  HGA)  quite  abstract  and  difficult  to  relate  to  their  day-to-day 
work  [13].  The  goal  and  error  states  of  variables  may  seem  especially  difficult  to  elicit  from 
operators  when  the  variables  themselves  are  qualitative  and  governed  by  intentional  (rather  than 
physical)  constraints.  Ultimately,  the  strengths  and  weaknesses  of  HGA  relative  to  previous 
techniques  may  depend  on  the  objectives  of,  and  the  constraints  imposed  on,  a  specific  analysis. 

Although  HGA  is  goal-based,  it  does  bear  similarities  as  well  as  differences  to  task-based  and 
function-based  approaches.  Like  task-based  approaches,  HGA  tends  to  view  the  system  from  the 
operators’  perspective.  HGA  asks  what  operators  need  to  monitor,  control,  and  achieve; 
and  task-based  approaches  ask  what  activities  operators  need  to  perform.  In  contrast, 
function-based  approaches  tend  to  view  the  system  from  the  designer/engineer’s  perspective,  by 
asking  what  the  system  is  designed  to  achieve.  In  other  words,  HGA  and  task-based  approaches 
take  a  psychological  perspective  to  analyzing  work,  while  function-based  approaches  take  a 
systems  perspective  [14], 

Nevertheless,  HGA  and  function-based  approaches  are  similar  in  that  they  identify  all  of  the  goals 
or  functions  for  the  entire  system  before  any  goal  or  function  is  assigned  to  any  operator.  In  fact, 
in  the  case  of  function-based  approaches,  there  is  no  (immediate  or  delayed)  requirement  to 
perform  any  operator  assignments.  For  both  HGA  and  function  -based  approaches, 
operators  can  be  human  or  automated  systems.  In  contrast,  task-based  approaches  tend  to  identify 
and  analyze  tasks  that  have  been  pre-assigned  specifically  to  human  operators  [4],  often  because 
they  are  conducted  to  inform  the  design  of  training  procedures  or  other  forms  of  support  for 
specific  operators. 

For  the  current  analysis  of  the  Halifax  Class  Frigate  operations  room,  HGA  was  chosen  as  an 
appropriate  method  of  analysis  for  two  reasons:  first  because  it  focused  on  the  operators  rather 
than  the  system;  and  second,  because  it  was  intended  to  accommodate  a  constantly  moving  target 
(cf.,  [15]).  In  terms  of  focus,  a  frigate’s  engineered  functions  (e.g.,  “to  float,  to  move,  to  fight”) 
and  the  physical  constraints  associated  with  these  functions  might  be  relatively  straightforward  to 
identify  using  a  function-based  approach.  However,  the  operations  room  was  specifically 
responsible  for  executing  the  crew’s  missions  (e.g.,  sea  denial,  peace  support,  humanitarian 
assistance)  [16],  and  as  such,  its  analysis  would  be  more  appropriately  focused  on  identifying 
what  goals  the  crew  needed  to  achieve  and  the  intentional  constraints  associated  with  these  goals. 
In  terms  of  accommodating  changes,  new  operator  positions  were  being  created  and  new 
automated  processes  were  being  introduced  in  the  operations  room  to  support  future  operations. 
The  flexibility  to  consider  re-allocation  of  roles  between  human  operators  and/or  automation  by 
re-using  and  adapting  significant  portions  of  the  original  analysis  was  therefore  both  necessary 
and  valued. 


1.3  Previous  application 

A  literature  survey  completed  as  part  of  this  project  showed  that  HGA  had  only  had  one  previous 
application,  where  an  HGA  was  conducted  to  support  the  design  of  intelligent  agents  to  aid  a 
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three-person  airborne  crew  in  the  control  of  UAVs  [8,  9],  In  that  application,  only  the  initial 
phases  of  HGA,  as  proposed  in  [1],  were  completed.  These  phases  included: 

-  the  development  of  a  goal  hierarchy, 

-  the  assignment  of  a  “controlled  variable”  to  each  goal,  and 

-  the  assignment  of  each  goal  to  an  operator,  that  may  be  human  or  machine. 

Since  the  objective  of  the  UAV  analysis  was  to  predict  the  impact  of  two  different  interfaces  (i.e., 
one  without  and  one  with  intelligent  agents),  two  sets  of  goal  assignments  were  made:  one  where 
each  goal  was  assigned  to  one  of  the  three  human  operators,  and  another  where  each  goal  was 
assigned  to  one  of  the  three  human  operators  or  to  an  intelligent  agent. 

There  were  several  key  differences  between  the  previous  UAV  analysis  and  the  current  frigate 
operations  room  analysis. 

-  First,  the  current  analysis  took  HGA  out  of  the  aviation  domain,  where  it  was  conceived 
and  first  applied,  into  the  maritime  domain; 

-  Second,  the  current  analysis  extended  beyond  the  initial  phases  of  HGA,  and  included  a 
stability  analysis  and  an  upward  flow  analysis  (to  be  described  in  detail  in  the  next  section); 

-  Third,  the  current  analysis  was  significantly  greater  in  scale  and  in  complexity,  considering 
the  possibilities  for  interactions  between  1 1  operators,  compared  to  3  operators  in 
the  UAV  analysis. 

Therefore,  the  current  analysis  was  an  important  step  forward  in  exploring  the  utility  of  the  HGA 
approach  for  analyzing  a  complex  military  system.  It  provided  an  opportunity  to  reflect  on  the 
success  factors,  limitations,  and  challenges  for  conducting  a  large-scale  HGA.  It  also  produced 
much-needed  examples  of  each  step  in  an  HGA  that  can  be  referenced  by  other  analysts  who  are 
interested  in  applying  HGA.  While  the  original  conceptual  paper  on  HGA  [1]  did  include 
examples,  they  were  hypothetical,  and  were  not  based  on  the  complete  analysis  of  an  actual 
system.  In  contrast,  this  paper  provides  a  real-world  case  study  to  demonstrate  each  step  in  an 
HGA,  including  a  stability  analysis  and  an  upward  flow  analysis,  both  of  which  had  not  been 
included  in  the  UAV  analysis. 

The  focus  of  this  paper  is  on  the  practice  rather  than  the  theory  of  HGA.  Therefore,  the  next 
section  contains  only  a  brief  summary  of  the  analytical  approach.  For  a  detailed  theoretical  and 
methodological  account  of  HGA,  the  reader  is  directed  to  [1]. 

1.4  Hierarchical  goal  analysis 

HGA  is  based  on  Perceptual  Control  Theory  [17],  which  posits  that  humans  operate  as 
perceptually  driven,  goal  referenced,  feedback  systems,  in  that  all  human  behaviours  are 
responses  to  errors,  or  differences,  that  are  perceived  between  current  states  of  the  world  and  goal 
states.  To  analyze  a  cognitive  system,  HGA  identifies  goals  that  serve  as  reference  points  for 
perception  (i.e.,  the  detection  of  error)  and  control  (i.e.,  the  correction  of  error).  Goals  are 
identified  through  a  process  of  hierarchical  decomposition,  starting  with  top-level  goals  that 
characterize  the  overall  system,  to  lower-level  goals  that  support  the  attainment  of  the  higher- 
level  goals.  Since  all  goals  in  an  HGA  are  reference  points  for  perception  and  control,  each  goal 
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can  be  considered  to  be  prefaced  by  the  phrase:  “I  want  to  perceive  that”.  Throughout  this  paper 
the  expression  “I  want  to  perceive  that”  is  represented  by  an  ellipsis  (...).  For  each  goal,  F1GA 
also  identifies  the  controlled  variable  whose  state  is  compared  to  the  goal  and  manipulated  as 
required  to  meet  that  goal. 

Once  all  goals  and  their  corresponding  controlled  variables  have  been  identified,  each  goal  is 
assigned  to  an  operator  in  the  system.  This  operator  can  be  human  or  automated.  Depending  on 
the  purpose  of  the  analysis,  operator  assignment  may  be  based  on  an  existing  system  (e.g.,  to 
identify  needs  or  opportunities  for  improvement)  or  on  an  envisioned  system  (e.g.,  to  consider 
different  alternatives  for  system  design).  Regardless,  every  goal  at  every  level  is  assigned  to  an 
individual  operator,  not  a  collection  of  operators.  The  same  operator  can  be  assigned  to  multiple 
goals,  at  high  and/or  low  levels  of  the  hierarchy,  along  one  or  multiple  branches  of  the  hierarchy. 
A  key  characteristic  of  F1GA  is  that  every  goal  at  every  level  of  the  hierarchy  is  analyzed  in  the 
same  way,  to  the  same  degree  of  detail.  Controlled  variables  are  identified  for  all  high  and 
middle-level  goals  (not  just  bottom-level  goals),  and  operator  assignments  are  made  for  all 
middle-level  goals  (not  just  top-level  goals  or  bottom-level  goals). 

Once  goal  assignment  is  complete,  two  types  of  follow-on  analyses  can  be  conducted:  a  stability 
analysis;  and  an  upward  flow  analysis.  A  stability  analysis  identifies  all  instances  where  the  same 
controlled  variable  is  associated  with  multiple  goals  that  are  assigned  to  different  operators, 
causing  potential  instability  in  the  system  as  operators  attempt  to  drive  the  variable  in  different 
directions.  In  essence,  a  stability  analysis  produces  control  requirements  that  specifically  point  to 
where  control  may  need  to  be  limited  to  specific  operators,  times  and/or  contexts.  The  second 
analysis,  an  upward  flow  analysis,  looks  at  each  lower-level  goal  and  determines  if  the  supported 
higher-level  goal  is  assigned  to  a  different  operator.  This  would  require  the  state  of  the  variable 
for  the  lower-level  goal  to  be  fed  back  to  the  operator  assigned  to  the  higher-level  goal.  In 
essence,  an  upward  flow  analysis  produces  feedback  requirements,  specifically  the  need  for 
information  to  be  transferred  between  operators  so  that  the  individual  responsible  for  the  upper- 
level  goal  can  maintain  awareness  of  the  state  or  progress  of  the  lower-level  goal.  The  form  (e.g., 
verbal  or  visual)  in  which  information  is  transferred  may  also  be  relevant  to  the  analysis. 


4 


DRDC  Toronto  TR  2007-161 


2  Method 


2.1  Halifax  class  frigate  operations  room 

The  Canadian  Navy’s  Halifax  Class  Frigates  were  launched  between  1992  and  1996.  They  are 
helicopter-carrying  warships  with  anti-submarine,  surface-to-surface,  and  surface-to-air 
capabilities.  It  is  important  for  the  design  of  the  frigate’s  operations  room  to  be  adapted  to  the 
needs  of  future  missions  -  from  the  physical  layout  of  the  operations  room,  to  the  individual 
workstations,  including  displays  and  controls,  to  the  decision  aiding  technologies  that  are 
available  on  these  workstations,  to  new  positions  that  may  be  added  to  the  operations  room  team. 
This  HGA  included,  but  was  not  be  limited  to,  1 1  positions  in  the  operations  room,  including  the 
Commanding  Officer  (CO)  of  the  frigate,  the  Operations  Room  Officer  (ORO),  the  Directors  of 
Above  Water  Warfare  (Sensor  Weapon  Controller  (SWC))  and  Under  Water  Warfare  (Assistant 
Sensor  Weapon  Controller  (ASWC))  who  supported  the  ORO,  and  their  key  support  staff 
including  the  Track  Supervisor  (TS),  the  Electronic  Warfare  Supervisor  (EWS),  the  Operations 
Room  Supervisor  (ORS),  the  Air  Raid  Report  Officer  (ARRO),  the  Anti-Submarine  Plotting 
Operator  (ASPO),  the  Information  Management  Director  (IMD)  and  Warfare  Duty  Officer. 
However,  since  the  HGA  approach  initially  required  identifying  goals  for  the  entire 
system  (i.e.,  the  operations  room)  some  goals  in  the  resulting  analysis  were  assigned  to  positions 
other  than  the  1 1  targeted.  Instances  of  these  additions  are  identified  throughout  the  paper,  and 
not  all  of  these  additional  operators  worked  within  the  operations  room.  Although  the  Navy 
engages  in  a  wide  range  of  operations,  a  “composite”  military  scenario  (i.e.,  one  that  included 
aspects  of  above  water  and  under  water  warfare)  was  used  as  the  basis  for  this  analysis  of 
the  operations  room  (see  Section  3.1). 

2.2  Goal  analysis 

The  HGA  began  with  identification  of  the  top-level  goals  of  the  operations  room.  The  core 
analysis  team  included  two  human  factors  experts  (one  an  experienced  ORO),  who  were 
supported  by  two  additional  domain  experts1  (an  ex-CO  and  an  Above  Water  Director).  Naval 
doctrine,  training  manuals,  standard  operating  procedures,  and  other  relevant  documentation  were 
reviewed  by  the  team  and  used  to  identify  the  top-level  goals,  which  were  subsequently 
decomposed  into  first-  and  second-level  goals.  The  decomposition  was  reviewed  for 
completeness  and  accuracy  at  this  point  by  two  representatives  of  the  Navy  (i.e.,  the  clients)  to 
ensure  that  the  analysis,  especially  its  scope  and  focus,  reflected  the  objective  of  supporting 
redesign  of  the  operations  room,  and  by  two  Defence  Scientists  to  ensure  that  the  analysis 
adhered  to  the  principles  of  HGA. 


1  It  is  important  to  distinguish  between  domain  experts  and  SMEs  in  this  study.  Domain  experts  refers 
specifically  to  the  two  experienced  operators  who  formed  part  of  the  analysis  team.  They  worked  hand-in- 
hand  with  the  human  factors  experts  over  an  extended  period  to  identify  and  analyze  the  goals  of  the 
operations  room.  These  two  domain  experts  were  different  from  the  36  SMEs  who  reviewed  the  outputs  of 
the  analyses  that  were  performed  by  the  domain  experts  together  with  the  human  factors  experts.  Aside 
from  two  review  sessions  scheduled  specifically  for  key  points  in  the  analysis  process,  there  was  no 
ongoing  interaction  between  the  SMEs  and  the  analysis  team  consisting  of  domain  experts  and  human 
factors  experts. 
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The  analysis  team  then  decomposed  the  goal  hierarchy  and  assigned  operators  to  the  goals  at  all 
levels.  A  validation  session  was  conducted  with  14  SMEs  who  were  serving  together  on  an 
operational  frigate  and  represented  all  but  one  of  the  targeted  positions.  The  review  of  the  goal 
hierarchy  proceeded  both  top-down  and  bottom-up.  The  top-down  approach  was  relatively 
structured;  the  SMEs  started  at  the  top  of  the  hierarchy  and  examined  each  level  for 
completeness,  and  for  accurate  connections  to  the  next  level.  The  bottom-up  approach  was  less 
structured;  the  SMEs  described  activities  they  performed  and  challenged  the  analysts  to  identify 
the  goal  that  captured  that  activity.  This  process  helped  to  confirm  that  the  goal  hierarchy  was 
complete  and  the  relationships  between  goals  were  accurate.  The  SMEs  also  reviewed  the 
operator  assignments  for  all  goals  as  recorded  by  the  analysts.  As  a  result  of  the  SME  session, 
appropriate  revisions  were  made  to  the  goal  hierarchy  and  to  the  operator  assignments.  The  most 
important  revision  was  the  creation  of  a  new  first-level  goal,  “...  predeployment  preparations  are 
complete”,  that  inherited  some  sub-goals  from  an  existing  first-level  goal. 

2.3  Stability  analysis  and  upward  flow  analysis 

The  analysis  team  used  the  revised  goal  hierarchy  and  operator  assignments  as  the  basis  for  a 
stability  analysis,  and  with  these  produced  a  list  of  controlled  variables  with  the  potential  for 
instability  (i.e.,  goal  conflict).  Each  of  these  variables  was  associated  with  two  or  more  goals  that 
could  cause  the  variable  to  be  driven  in  different  directions.  The  operators  associated  with  these 
goals  were  identified,  as  was  a  list  of  all  possible  combinations  of  these  operators.  For  each 
variable  with  the  potential  for  instability,  at  least  one  method  for  amelioration  was  recommended 
by  the  analysis  team.  For  example,  the  variable  “ship’s  heading”  was  determined  to  be  a  source  of 
possible  conflict  because  it  related  to  both  above  water  as  well  as  under  water  warfare. 
Consequently,  tactics  could  dictate  requirements  for  more  than  one  course.  A  corrective 
recommendation  proposed  was  that  all  orders  for  the  ship’s  course  must  go  through  the  ORO 
when  there  are  simultaneous  threats  from  above  water  and  under  water.  The  resolution  of  goal 
conflicts  can  be  considered  as  an  application  of  the  stability  analysis  results,  rather  than  a  step 
within  the  analysis  itself. 

The  analysis  team  also  used  the  revised  goal  hierarchy  and  operator  assignments  to  conduct  an 
upward  flow  analysis.  The  process  began  with  a  list  of  all  goals  excluding  the  highest  level  goals. 
For  each  goal,  the  assigned  operator  was  identified,  along  with  the  assigned  operators  for  its 
parent  goal.  For  every  case  where  a  goal  and  its  sub-goal  were  assigned  to  different  operators 
(i.e.,  there  was  a  requirement  for  feedback),  the  current  method  for  providing  feedback  was 
identified.  The  analysis  team  then  went  beyond  a  basic  upward  flow  analysis  by  rating,  based  on 
their  operational  experience,  each  feedback  requirement  as  important  or  not  important,  and  as 
frequent  or  not  frequent.  For  each  feedback  requirement,  they  also  categorized  the  current  method 
for  providing  feedback  as  visual  (e.g.,  reading  text  or  graphics,  observing  activities)  or  verbal 
(e.g.,  listening  to  reports  or  alarms).  Finally,  for  each  pair  of  operators  (i.e.,  the  1 1  targeted 
operators  could  be  arranged  into  110  possible  pairs),  the  total  number  of  feedback  requirements 
was  computed,  as  well  as  the  number  of  important  requirements,  the  number  of  frequent 
requirements,  and  the  numbers  of  requirements  addressed  verbally  versus  visually. 

The  results  of  the  stability  analysis  and  the  upward  flow  analysis  were  validated  in  a  second  SME 
session,  involving  22  SMEs  who  were  part  of  the  ship’s  company  on  an  operational  high¬ 
readiness  frigate.  In  this  session,  SMEs  represented  all  but  two  of  the  targeted  positions.  For  the 
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stability  analysis,  the  SMEs  reviewed  the  sources  of  instability  and  augmented  the  list  of 
proposed  amelioration  techniques  for  stable  control.  For  the  upward  flow  analysis,  the  SMEs  also 
reviewed  the  feedback  requirements  and  their  categorizations  (i.e.,  importance,  frequency,  type). 
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3  Results 


3.1  Goal  analysis 

The  HGA  identified  three  top-level  goals  for  the  Halifax  Class  Frigate  operations  room: 

1 .  ...  use  of  the  sea  is  denied  to  enemy  forces; 

2.  ...  assistance  is  provided  to  other  government  departments;  and 

3.  ...  peace  support  operations  are  conducted. 

These  goals  corresponded  respectively  to  the  military,  constabulary,  and  diplomatic  roles  that  are 
held  by  the  Navy  across  time  and  across  ships  [16].  Each  of  these  top-level  goals  was 
decomposed  into  seven  first-level  goals.  Reviews  by  stakeholders  and  by  SMEs  confirmed  that 
the  decomposition  of  goals  from  the  top-level  to  the  first  and  second  levels  were  identical  across 
the  three  top-level  goals.  Therefore,  the  decision  was  made  to  only  decompose  the  military  goal  in 
full.  The  top  and  first  levels  of  the  military  goal  are  shown  in  Figure  1. 


Figure  1:  Top  and  first  levels  of  the  military  goal. 


Figure  2  shows  decomposition  of  the  goal  “...  an  optimal  level  of  situation  awareness  is  being 
maintained”  to  the  second  level.  Table  1  displays  in  tabular  form  the  successive  decomposition 
of  one  of  these  second-level  goals  “...  an  accurate  Recognized  Maritime  Picture  (RMP)  is 
created  and  maintained”  to  the  lowest  level  by  showing  the  decomposition  of  one  sub-goal  at 
every  level.  Table  1  also  shows  the  controlled  variable  and  the  assigned  operator  for  each  goal. 
The  full  goal  hierarchy  included  563  goals  and  ranged  from  four  to  nine  levels  deep.  It  is 
shown  in  a  tabular  format  in  [18],  along  with  563  one-page  templates  that  are  populated  with 
the  attributes  for  each  goal.  In  this  paper,  only  the  most  significant  results  related  to  the  goal 
hierarchy  and  the  assigned  operators  are  summarized  below. 
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Figure  2:  Partial  decomposition  of  the  situation  awareness  goal 

All  of  the  1 1  targeted  operators  were  assigned  to  goals  at  multiple  levels  of  the  hierarchy.  Ten 
percent  of  the  goals  (i.e.,  56)  were  assigned  to  operators  other  than  the  1 1  who  were  targeted.  Of 
these  other  operators,  the  SAC  and  the  OOW  were  both  assigned  to  the  largest  number  of  goals 
(i.e.,  16).  Since  the  SAC  was  a  member  of  the  operations  room,  the  subsequent  analyses  that 
were  focusing  on  stability  and  upward  information  flow  were  extended  to  include  the  SAC.  The 
OOW  did  not  work  within  the  operations  room  so  was  excluded  from  subsequent  analyses. 

Of  the  1 1  targeted  operators,  the  ORO  (assigned  to  86  goals),  the  SWC  (assigned  to  1 1 1  goals) 
and  the  ASWC  (assigned  to  76  goals)  accounted  for  nearly  half  the  goals  (i.e.,  273/563  or  48%). 
They  were  assigned  to  goals  ranging  from  as  high  as  the  first  level  to  as  low  as  the  fifth  level  for 
the  ORO,  or  the  eighth  level  for  the  two  directors  (SWC  and  ASWC).  Goals  assigned  to  the  ORO 
were  decomposed  from  six  of  the  seven  first-level  goals,  and  goals  assigned  to  the  two 
directors  were  decomposed  from  five  of  the  seven  first-level  goals.  In  contrast,  the  CO  was  only 
assigned  to  34  goals,  but  these  included  all  seven  first-level  goals.  Note  that  the  total 
number  of  goals  assigned  to  an  operator  is  not  a  direct  indication  of  workload.  For  example,  the 
SAC  was  allocated  only  16  goals,  but  when  controlling  aircraft,  this  operator  could 
experience  a  high  attentional  demand.  However,  the  number  of  goals  is  related  to  the  breadth  of 
responsibility  assigned  to  an  operator  and  the  knowledge  and  experience  that  would  be  required 
to  achieve  these  goals. 

During  the  analysis  the  domain  experts  determined  that  in  the  frigate  operations  room  every  goal 
should  actually  be  assigned  to  a  primary  operator,  and  to  a  secondary  operator  who  was  expected 
to  take  over  responsibility  for  a  goal  if  the  primary  operator  was  otherwise  engaged.  Results 
discussed  so  far  have  pertained  only  to  the  assignment  of  the  primary  operator.  The  following 
discussion  of  the  stability  analysis  and  the  upward  information  flow  analysis  considers  both 
primary  and  secondary  operators. 
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Table  1:  Subset  of  the  decomposition  of  the  Recognized  Maritime  Picture  (RMP)  goal. 


Level 

Goal 

Number 

Goal 

Controlled 

Variable 

Assigned 

Operator 

Sub-Goals 

2 

5.1 

...an  accurate  RMP  is 
created  and  maintained 

Currency  of  the 
RMP 

ORO 

5.1.1 

...an  accurate  tactical  air  picture 
is  compiled 

5.1.2 

...an  accurate  tactical  surface 
picture  is  compiled 

5.1.3 

...an  accurate  subsurface  picture 
is  compiled 

3 

5.1.2 

...an  accurate  tactical 
surface  picture  is 
compiled 

Status  of  the 
tactical  surface 
picture 

swc 

5. 1.2.1 

...coordination  with/of  other  units 
is  effective 

5. 1.2.2 

...  tactical  displays  are  optimized 

5. 1.2.3 

...effective  surface  track 
management 

5. 1.2.4 

...effective  visual  watch  is 
maintained  to  support  tactical 
surface  picture 

5. 1.2.5 

...effective  use  of  sensors  to 
compile  a  surface  picture 

5. 1.2.6 

...information  is  gathered  through 
the  use  of  ship-ship  warnings 

4 

5. 1.2.3 

...effective  surface  track 
management 

Status  of  surface 
track  management 

TS 

5. 1.2.3. 1 

...proper  application  of  criteria  to 
new  surface  contacts 

5.1.2.3.2 

...existing  tracks  are  updated 

5. 1.2.3. 3 

...reports  are  made  to  support  the 
tactical  surface  picture 

5 

5.1.2.3.2 

...existing  tracks  are 
updated 

Status  of  surface 
track 

TS 

5. 1.2. 3. 2.1 

...manual  updates  are 
successfully  executed 

5.1.2.3.2.2 

...automatic  updates  are 
occurring 

6 

5. 1.2. 3. 2.1 

...manual  updates  are 
successfully  executed 

Status  of  surface 
track 

TS 

3.2  Stability  analysis 

As  mentioned  previously,  the  goal  hierarchy  included  563  goals.  Each  goal  controlled  a  variable, 
and  each  goal  was  assigned  to  an  operator.  The  stability  analysis  identified  17  variables  as 
potential  sources  of  instability  (i.e.,  goal  conflict)  because  they  were  controlled  by  multiple 
operators  to  achieve  multiple  goals.  Table  2  shows  the  detailed  analysis  of  one  of  these  variables. 
These  variables  were  controlled  by  2  to  1 1  different  operators.  The  same  operator  often  controlled 
the  same  variable  for  multiple  goals.  In  one  case,  a  variable  was  controlled  by  9  operators,  and 
one  of  the  operators  controlled  that  variable  for  17  different  goals.  While  there  were  also  cases 
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where  each  operator  controlled  the  potentially  unstable  variable  for  only  a  single  goal,  most  of  the 
potentially  unstable  variables  were  associated  with  multiple  goals  per  operator. 


Table  2:  Sample  stability  analysis. 


Influenced 

Variable 

(External) 

Operator 
or  system 
Assign¬ 
ment 

Goal# 

Goal  /  Objective  /  Reference 

Potential 
for  Simul¬ 
taneous 

Control 

For  Stable 
Control 

Ship's  heading 
and  speed 

ASWC 

5.1.3.6.3 

...sensor  depth  is  monitored/adjusted 

ASWC 
and  SWC 

-all  demands  for 
ship's  course  or 
speed  must  go 
through  the  CO  or 
ORO 

WO 

6. 1.2.2. 1.1.1 

...assigned  sectors/station  maintained 

ASWC 
and  EWS 

WO 

6. 1.2.2. 1.1. 2 

...due  regard  for  sensor  limitations 

SWC  and 
ORO 

WO 

6. 1.2.2. 1.1.3 

...due  regard  for  navigational  safety 

ASWC 
and  ORO 

-CO  and  ORO 
must  remain  co¬ 
located  to  prevent 
simultaneous 
control  demands 

WO 

6. 1.2.2. 1.1.4 

...due  regard  for  engineering  status 

CO  and 
ORO 

WO 

6. 1.2.2. 1.1. 5 

...territorial  boundaries  are  observed 

OOW  and 
ORO 

ORO 

6.2.1. 1 

...assigned  sectors/station  maintained 

WO  and 
ORO 

ASWC 

6.2.1.2 

...due  regard  for  sensor  limitations 

WO  and 
CO 

-only  one  of  the 

CO  or  ORO  should 
have  control  of 
ship's  course  and 
speed,  this  is 
typically  the  ORO 

ORO 

6.2.1.3 

...due  regard  for  navigational  safety 

CO 

6.2.1.4 

...due  regard  for  engineering  status 

ORO 

6.2.1.5 

...territorial  boundaries  are  observed 

ORO 

6. 2.2. 1.1 

...helicopter  is  at  appropriate  status 

EWS 

6.5.1 . 1 .2.3.4 

...the  ship  is  effectively  manoeuvred 

-OOW  retains 
safety 

responsibility 

SWC 

6.5. 1.3.2 

...appropriate  positioning  to  support 
consorts 

SWC 

6. 5.2.2. 3 

...ship  is  optimally  positioned  for  offence 

-relationship 
between  the 

Warfare  Officer 
and  ship's  officers 
will  have  to  be 
worked  out  in 
advance  of  entering 
area  of  conflict 

SWC 

6. 5. 2.2. 4. 6.1 

...an  effective  application  of  tactics 

ASWC 

6.5.3. 1.3 

...defensive  tactics  are  appropriately 
conducted 

ASWC 

6.5.3. 1.3.1 

...appropriate  course/helm  and  speed  are 
chosen 

ASWC 

6. 5. 3.2. 1.1 

...effective  manoeuvring  for  offence 

ASWC 

6. 5. 3. 2.2. 1.1 

...assigned  sectors/station  maintained 

ASWC 

6. 5. 3. 2.2.2 

...search  effectively  executed 

ASWC 

6. 5. 3. 2.2. 2.1 

...assigned  sectors/station  maintained 

ASWC 

6. 5. 3. 3.4 

...assets  are  effectively 
stationed/ employed 

ORO 

6.7.4 

...ship  is  effectively  conducting  the  search 

ORO 

6.8 

...emergencies  are  responded  to  efficiently 
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For  the  17  potential  sources  of  instability,  the  number  of  recommendations  for  stable  control 
ranged  from  two  to  five.  Some  recommendations  pertained  to  workspace  design  (e.g.,  co-location 
of  specific  operators).  Others  pertained  to  technologies  including  computer  support  for 
cooperative  work  (e.g.,  text-messaging  between  operators),  interface  design  for  monitoring  (e.g., 
shared  displays)  and  for  control  (e.g.,  lock-out  of  specific  operators).  Still  others  pertained  to 
processes  (e.g.,  all  requests  of  a  specific  type  to  go  through  a  specific  operator). 

A  table  similar  to  Table  2  was  developed  for  each  of  the  17  sources  of  potential  stability.  These 
tables  can  be  found  in  [18]. 

3.3  Upward  flow  analysis 

The  upward  flow  analysis  identified  878  instances  where  an  operator  assigned  to  a  higher-level 
goal  required  feedback  on  a  lower- level  goal.  The  domain  experts  rated  518  of  these  feedback 
requirements  as  important,  and  182  as  frequent.  The  analysis  also  found  that  currently  38 
requirements  were  primarily  being  satisfied  visually,  and  248  were  primarily  being  satisfied 
verbally.  Considering  all  possible  pairs  of  the  12  operators  (including  the  SAC),  the  highest 
overall  requirement  for  feedback  was  found  between  the  SWC  and  the  ORO  (i.e.,  1 12  instances), 
followed  by  the  ORO  and  the  CO  (i.e.,  81  instances),  and  the  ASWC  and  the  ORO  (i.e.,  78 
instances).  Fifty-one  percent  of  the  requirements  between  these  three  pairs  of  operators  were 
addressed  verbally  rather  than  visually.  The  highest  number  of  important  feedback  requirements 
were  found  between  the  SWC  and  the  ORO  (i.e.,  66  requirements),  and  58%  of  these  were 
addressed  verbally.  The  highest  number  of  frequent  feedback  requirements  were  found  between 
the  SWC  and  the  ORO  (i.e.,  25  requirements),  and  60%  of  these  were  addressed 
verbally.  Figure  3  provides  a  three-dimensional  view  of  the  overall  numbers  of  feedback 
requirements  between  the  targeted  operators.  Figure  4  shows  only  those  requirements  that  are 
currently  addressed  verbally. 

Figures  3  and  4  are  screen  shots  from  a  three-dimensional  viewer  that  had  been  developed 
specifically  at  the  request  of  the  Navy  to  enable  the  visualization  and  comparison  of  feedback 
requirements  between  different  pairs  of  operators.  The  viewer  provides  controls  along  the  left  and 
the  bottom  of  the  screen  that  enable  the  user  to  view: 

-  the  feedback  requirements  associated  with  one  or  more  of  the  seven  top-level  goals; 

-  the  feedback  requirements  considering  only  the  primary,  only  the  secondary,  or  both 
operator(s)  assigned  to  each  goal; 

-  the  feedback  requirements  that  have  been  rated  by  domain  experts  (and  verified  by  SMEs) 
as  frequent,  as  important,  or  all  requirements  regardless  of  ratings; 

-  the  feedback  requirements  that  were  addressed  verbally  or  visually  or  both;  and 

-  the  feedback  requirements  in  absolute  numbers,  or  as  percentages  of  the  overall  number  of 
requirements. 
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Figure  3:  Overall  number  of  feedback  requirements. 
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Figure  4:  Numbers  of  feedback  requirements  addressed  verbally. 
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4  Discussion 


Development  of  the  goal  hierarchy  revealed  that  there  was  substantial  variation  in  the  number  of 
goals  assigned  to  each  operator  (e.g.,  1 1 1  goals  for  the  SWC  compared  to  16  goals  for  the  SAC, 
and  compared  to  an  estimated  average  of  approximately  51  goals  per  operator  based  on  the  total 
number  of  goals  and  targeted  operators).  Operators  in  the  most  senior  positions  were  not 
necessarily  assigned  to  more  goals  than  their  subordinates  (e.g.,  the  ORO  had  86  goals  compared 
to  the  CO’s  34;  the  SWC  had  111  goals  compared  to  the  ORO’s  86).  Nor  were  senior  positions 
assigned  only  to  high-level  goals  (e.g.,  the  SWC  and  the  ASWC  were  assigned  to  some  eighth- 
level  goals).  Although  the  analysts  targeted  select  operators  within  a  system,  the  HGA  naturally 
revealed  additional  operators  who  should  be  included  in  the  analysis  if  an  understanding  of  the 
overall  system  is  desired  (e.g.,  the  SAC  and  the  OOW  were  both  assigned  to  goals  that  provided 
direct  or  indirect  support  to  the  high-level  goals  of  the  operations  room). 

In  other  words,  HGA  did  not  take  an  organization-based  or  position-based  approach  to  analyzing 
a  sociotechnical  system.  In  reality,  the  number  of  operators  in  a  system,  how  they  are  organized 
into  teams,  and  to  whom  they  report  should  be  the  end  points  (i.e.,  solutions)  rather  than  the  start 
points  to  a  systems  design  problem.  When  an  organization-based  or  position-based  approach  is 
used  to  analyze  a  cognitive  system,  these  specifics  are  pre-defmed,  and  all  that  remains  to  be 
analyzed  is  how  the  performance  of  specific  operators  in  their  pre-specified  roles  can  be 
supported  and/or  measured.  But  truly  innovative  system-wide  solutions  such  as  changes  to  the 
command  or  reporting  structure,  addition  or  removal  of  operators,  or  re-distribution  of  authority 
and  responsibility  between  operators  become  difficult  if  not  impossible  to  consider  through 
organization-  or  position-based  approaches.  Through  these  approaches  it  would  not  be  clear  how 
the  requirements  of  one  position  would  change  if  changes  were  made  to  any  or  all  of  the  other 
positions.  By  first  identifying  all  of  the  goals  that  need  to  be  achieved  (rather  than  all  of  the 
operators  in  the  system)  and  defining  the  relationships  between  these  goals  (rather  than  between 
the  operators),  HGA  provided  a  solid  foundation  upon  which  different  system  design 
options  (i.e.,  different  goal  assignments)  might  be  considered,  without  the  need  to  re-analyze  the 
entire  system  as  soon  as  any  assumption  about  an  operator’s  role  was  violated. 

Assumptions  about  an  operator’s  role  (i.e.,  goal  assignments)  had  to  be  made  before  proceeding 
to  the  stability  analysis  and  the  upward  flow  analysis.  But  it  would  certainly  be  possible  and 
likely  useful  to  apply  different  goal  assignments  (e.g.,  representing  an  existing  system  vs.  a  new 
system;  representing  two  different  designs  of  a  new  system)  and  to  compare  the  stability  and 
upward  flow  results  based  on  these  different  assignments  to  help  choose  the  optimal  design 
option.  In  this  application,  only  one  set  of  goal  assignments  was  made  based  on  the  existing 
system,  but  it  provided  benchmarks  to  which  design  options  being  developed  for  the  operations 
room  could  be  compared. 

4.1  Stability  analysis 

One  benchmark  resulting  from  the  stability  analysis  related  to  how  a  new  operations  room  could 
be  considered  an  improvement  over  the  existing  operations  room.  Since  the  existing  operations 
room  had  17  variables  that  could  contribute  to  instability  because  they  were  under  the  control  of 
multiple  operators,  the  new  operations  room  should  have  fewer  than  17  variables  with  the 
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potential  for  instability.  For  example,  one  target  for  the  new  operations  room  design  might  be  to 
achieve  a  50%  reduction  in  the  number  of  potentially  unstable  variables  (i.e.,  to  8  or  fewer 
variables)  by  re-assigning  goals  between  operators.  Moreover,  without  changing  the  goal 
assignments,  results  of  the  stability  analysis  could  still  support  options  analysis:  first,  each 
recommendation  for  improving  stability  could  be  rated  in  terms  of  the  number  of  variables  it 
would  help  to  stabilize  or  the  number  of  operators  it  would  help  to  de-conflict,  which  could  be 
easily  tabulated  from  results  in  the  form  shown  in  Table  2.  Then,  each  design  option,  embodying 
some  specific  combination  (but  not  all)  of  the  recommendations,  could  be  given  an  aggregated 
rating  and  a  ranking  relative  to  the  other  options.  It  would  also  be  possible  to  attach  costs  to  the 
recommendations  and  to  aggregate  these  costs  to  produce  the  total  costs  for  complete  design 
options.  A  cost-benefit  analysis  could  then  be  conducted,  once  again  using  the  number  of 
variables  and/or  operators  affected  to  measure  the  benefit  associated  with  each  option. 

Results  of  a  stability  analysis  could  also  support  risk  analysis.  For  example,  each  recommendation 
for  ameliorating  instability  could  be  categorized  based  on  the  expected  effect  of  neutralizing  the 
instability  and  weighted  for  comparison.  Categories  might  be:  eliminating  instability  (lowest 
risk),  reducing  instability  (medium  risk),  or  discouraging  instability  (highest  risk).  A 
recommendation  that  makes  it  absolutely  impossible  for  multiple  operators  to  access  the  same 
variable  at  the  same  time  would  be  considered  to  eliminate  instability.  Generally,  technological  or 
system  design  interventions  would  be  used  to  eliminate  instability.  One  such  recommendation 
from  the  current  application  was  for  the  system  to  lock  out  non-critical  users  of  a  radar  during 
weapon  engagement.  A  recommendation  would  be  considered  to  reduce  instability  if  it  makes  it 
difficult  for  multiple  operators  to  access  the  same  variable  at  the  same  time.  For  example,  in  the 
current  application  one  recommendation  was  for  all  changes  to  the  engineering  state  to  go 
through  the  ORO  to  the  OOW  and  then  the  CO.  While  it  is  physically  possible  for  someone  to 
request  or  to  initiate  a  change  without  seeking  the  required  approval,  it  is  highly  unlikely  for  the 
change  to  be  accepted  or  implemented  in  full.  Consequently,  training  or  process  interventions 
would  generally  be  necessary  to  reduce  instability.  Finally,  a  recommendation  that  makes  it 
unlikely  for  multiple  operators  to  try  to  access  the  same  variable  at  the  same  time  would  be 
considered  to  discourage  instability.  One  such  recommendation  from  the  current  application  was 
the  co-location  of  the  two  directors  to  make  it  easier  for  them  to  detect  and  immediately  resolve 
any  conflict,  before  either  one  tries  to  control  a  variable  in  a  way  that  is  inconsistent  with  what  the 
other  wishes  or  has  agreed  to  do.  Generally,  design  of  the  workspace  or  of  collaborative  or 
communication  aids  would  be  used  to  discourage  instability.  To  conduct  a  risk  analysis,  a 
weighting  factor  could  be  assigned  to  each  category  of  recommendations  to  represent  its  level  of 
risk.  An  overall  level  of  risk  could  then  be  computed  for  an  overall  operations  room  design 
(defined  as  specific  combinations  of  the  high,  medium,  and  low  risk  recommendations).  An 
overall  design  (including  technological,  process  and  other  aspects)  could  then  be  compared 
against  alternate  overall  designs. 

4.2  Upward  flow  analysis 

One  important  finding  from  the  upward  flow  analysis  was  that  the  design  of  the  existing 
operations  room  imposed  much  greater  demands  on  the  operators’  verbal  channel  than  on  their 
visual  channel  (e.g.,  58%  of  the  communication  requirements  between  the  ORO  and  the  SWC 
were  addressed  verbally).  Therefore,  the  new  operations  room  design  might  consider  increasing 
the  operators’  capability  to  give  and  receive  feedback  visually  through  the  design  of  individual 
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and/or  shared  displays,  or  through  the  co-location  of  operators  such  that  they  can  better  observe 
one  another’s  activities.  It  might  also  be  useful  to  explore  other  non-verbal  modes  of  feedback 
such  as  tactile  feedback,  since  all  feedback  requirements  were  addressed  visually  or  verbally  in 
the  existing  operations  room. 

The  upward  flow  analysis  identified  additional  opportunities  for  improving  the  design  of  the 
existing  operations  room,  if  the  design  goal  was  to  optimize  the  room  layout  based  on  the 
requirement  for  feedback  between  operators.  First,  in  the  existing  design,  some  operators  were 
positioned  adjacent  to  each  other  although  they  had  a  very  low  requirement  to  provide  feedback 
to  each  other  (e.g.,  the  SWC  and  the  ASWC  sat  side  by  side,  but  they  shared  only  one  feedback 
requirement).  On  the  other  hand,  some  operators  were  separated  when  they  had  a  high 
requirement  to  provide  feedback  to  each  other  (e.g.,  the  SWC  sat  between  the  ORO  and  the 
ASWC,  separating  them,  but  the  ASWC  actually  shared  78  feedback  requirements  with  the 
ORO).  There  were  other  examples  where  the  physical  proximity  between  two  operators  did  not 
correspond  to  the  requirement  for  feedback  between  these  operators.  For  instance,  the  ASPO  sat 
in  the  front  row  of  the  operations  room  between  the  SAC  and  the  TS,  even  though  they  shared 
zero  and  one  feedback  requirement,  respectively.  The  ASPO  actually  shared  the  highest  number 
of  feedback  requirements  (i.e.,  18)  with  the  ASWC,  but  they  sat  in  separate  rows,  and  the  ASPO 
actually  faced  away  from  the  ASWC.  In  fact,  the  ASPO’s  position  was  closer  to  the  SWC  (with 
whom  the  ASPO  shared  no  feedback  requirement)  than  to  the  ASWC.  Of  course,  it  was  quite 
possible  that  the  existing  operations  room  had  been  designed  with  a  different  objective  in  mind 
(e.g.,  perhaps  to  locate  operators  based  on  the  command  structure  rather  than  on  communication 
need,  when  voice  nets  provided  an  alternative  to  face-to-face  communication).  Ultimately,  the 
relative  merits  of  different  design  options  would  depend  on  the  criteria  applied. 

In  any  case,  if  supporting  communication  between  operators  was  the  design  objective,  then  one 
reasonable  application  of  the  upward  flow  analysis  results  would  be  to  develop  design  options 
that  would  place  operators  who  shared  the  highest  number  of  feedback  requirements  closest 
together  and  operators  who  shared  the  lowest  number  of  feedback  requirements  furthest  apart. 
Admittedly,  it  could  be  very  challenging  to  place  some  operators  who  shared  a  large  number  of 
feedback  requirements  with  many  others,  but  the  importance  and  the  frequency  of  these 
requirements  could  provide  additional  help  in  prioritizing  the  needs  for  proximity.  When 
generating  a  new  layout  for  the  operations  room,  instead  of  placing  the  operators  one-by-one,  it 
might  be  useful  to  perform  a  cluster  analysis  based  on  the  feedback  requirements,  to  identify 
logical  groupings  of  the  operators  (e.g.,  into  rows,  cells).  It  might  even  be  possible  to  optimize 
the  physical  distance  between  these  groupings. 

In  addition  to  using  the  upward  flow  analysis  results  to  generate  new  layouts  for  the  operations 
room,  the  results  could  be  used  to  evaluate  alternate  layouts,  such  as  giving  credit  to  layouts 
where  positions  with  high  communication  needs  were  oriented  so  that  they  faced  each  other.  To 
conduct  the  evaluation  it  would  be  important  to  pre-determine  the  primary  (versus  secondary  or 
even  tertiary)  basis  for  the  assessments.  For  example,  the  evaluator(s)  might  emphasize  total 
feedback  requirements  over  important  feedback  requirements  and  important  feedback 
requirements  over  frequent  feedback  requirements.  However,  a  different  set  of  criteria  might  be 
just  as  reasonable. 
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4.3  Optional  analyses 

Stability  analysis  and  upward  flow  analysis  were  just  two  of  many  different  types  of  analyses  that 
could  be  performed  using  HGA  as  the  basis.  In  particular,  these  two  types  of  analyses  required  as 
inputs:  1)  the  goal  hierarchy,  2)  the  controlled  variable  for  each  goal,  and  3)  the  assigned  operator 
for  each  goal.  Hendy  et  al  [1]  had  proposed  a  template  to  capture  other  attributes  for  each  goal, 
including:  required  knowledge  states  (declarative  and  situational),  perceptual/cognitive  processes, 
initiating  conditions,  ending  conditions,  input  sensation,  output  behaviour,  input  interface  and 
output  interface.  Once  elicited  from  domain  experts  and  validated  by  SMEs,  these  additional 
attributes  could  serve  as  inputs  to  other  follow-on  analyses,  including  but  not  limited  to: 

-  Training  needs  analysis,  primarily  based  on  the  required  knowledge  states  but  possibly  also 
the  initiating  and  ending  conditions,  and  the  input  and/or  output  interfaces; 

-  Manpower  analysis,  primarily  based  on  the  required  knowledge  states  but  possibly  also  on 
the  output  behaviours; 

-  Workflow  analysis  possibly  by  developing  a  task  network  simulation,  likely  based  on  the 
initiating  conditions,  ending  conditions  and  the  perceptual/cognitive  processes;  and 

-  Workload  analysis  possibly  by  using  a  task  network  simulation,  likely  based  on  the  input 
sensations  and  the  perceptual/cognitive  processes. 

In  the  current  case  study,  the  template  proposed  by  Hendy  et  al  [  1  ]  was  used  to  capture  additional 
attributes  for  each  goal,  and  the  gathered  information  was  used  to  develop  a  task  network 
simulation  to  analyze  workflow  and  workload.  These  additional  analyses  were  beyond  the  scope 
of  this  paper  given  its  focus  on  HGA  as  a  requirements  analysis  approach.  However,  details  on 
the  task  network  simulation  can  be  found  in  [18]. 

4.4  Lessons  learned  and  future  work 

The  current  application  of  HGA  benefited  tremendously  from  the  participation  of  two  domain 
experts  on  the  analysis  team  and  of  two  large  teams  of  SMEs  as  reviewers  at  key  stages  of  the 
HGA  process  (i.e.,  upon  completion  of  the  goal  hierarchy  and  operator  assignments;  and  upon 
completion  of  the  stability  analysis  and  upward  flow  analysis).  However,  both  the  analysis  team 
and  the  SMEs  found  the  HGA  methodology  to  be  difficult  to  learn,  primarily  because  they  were 
much  more  accustomed  to  thinking  in  terms  of: 

1 .  tasks  rather  than  goals; 

2.  specific  operator  positions  rather  than  the  overall  system;  and 

3.  a  timeline  of  activities  to  conduct  a  mission  rather  than  a  set  of  criteria  to  evaluate  mission 
effectiveness. 

For  operators,  most  of  the  training,  exercises,  and  operations  they  have  been  exposed  to  have 
been  structured  in  terms  of  tasks,  positions,  and  timelines.  While  the  learning  presented  some 
challenges,  the  SME  sessions  were  completed  as  scheduled  (i.e.,  three  work  days  for  the  first 
session,  and  three  work  days  for  the  second  session),  and  the  overall  HGA  was  completed  with  in  - 
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budget  and  on-time  (i.e.,  750  person-hours  over  two  months).  However,  given  the  iterative  nature 
of  the  analysis,  the  goal  hierarchy  was  revised  during  the  entire  duration  of  the  project. 

Future  applications  could  benefit  from  a  carefully  designed,  tested,  and  standardized  training 
program  on  the  HGA  methodology,  both  for  analysts  and  for  SMEs.  The  training  program  could 
be  paper-based  or  computer-based,  and  could  take  the  form  of  tutorials,  with  exercises  and  tests. 
Specifically,  the  training  should  focus  on: 

1 .  the  definition  of  a  goal  as  distinguished  from  a  task  or  a  function; 

2.  the  decomposition  of  a  goal,  as  distinguished  from  the  step-by-step  description  of  a  process; 

3.  the  identification  of  the  controlled  variable  for  a  given  goal,  as  distinguished  from  the 
identification  of  all  variables  relevant  to  a  goal  or  all  variables  controlled  by  all  sub¬ 
goals;  and 

4.  the  assignment  of  an  operator  to  a  goal,  as  distinguished  from  a  mapping  of  the 
goals  to  the  organizational  structure,  or  a  listing  of  all  operators  who  contribute  directly  and 
indirectly  to  a  goal. 

Future  applications  could  also  benefit  from  the  development  of  a  re-usable  HGA  tool  for 
documenting  and  manipulating  the  HGA  results,  and  to  support  the  development  and  evaluation 
of  alternate  design  options.  Specifically,  the  tool  should  provide  templates  for  the  analysts  to 
store  the  goals  as  entities  and  the  relationships  between  goals,  and  automatically  generate  the  goal 
hierarchy  based  on  these  entities  and  relationships.  For  each  goal,  the  tool  should  capture,  as  a 
minimum,  the  controlled  variable  and  a  “baseline”  operator  assignment.  The  tool  should  also 
enable  the  saving  of  alternate  operator  assignments.  Based  on  the  stored  information,  the  tool 
should  automatically  produce  results  of  the  stability  analysis  and  the  upward  flow  analysis,  based 
on  whichever  set  of  operator  assignments  was  chosen  by  the  analyst/designer.  An  even  more 
ambitious  tool  would  include  capability  to  capture  the  rating  and/or  weighting  of  different  control 
requirements  (from  the  stability  analysis)  and  feedback  requirements  (from  the  upward  flow 
analysis),  and  to  document,  tabulate,  and  compare  the  costs  and/or  benefits  of  different  design 
options  that  address  these  requirements. 

Considering  the  complexity  of  the  Halifax  Class  Frigate  operations  room  (i.e.,  more  than 
1 1  operators  controlling  93  different  external  variables  for  563  goals),  the  potential  for  instability 
was  relatively  low  (i.e.,  only  17  or  18%  of  the  variables  were  under  the  control  of  multiple 
operators).  While  the  clients  were  appreciative  of  the  stability  analysis  outputs  and 
recommendations,  they  observed  that  the  existing  operations  room  had  been  designed  to  minimize 
instability.  On  the  other  hand,  the  requirement  for  feedback  was  relatively  high  (i.e.,  878 
instances  where  feedback  on  one  goal  needs  to  be  given  to  an  operator  of  another  goal),  and  the 
application  of  these  results  was  of  keen  interest  to  the  clients.  Therefore,  another  key  success 
factor  for  this  application  of  HGA  was  the  development  of  a  three-dimensional  viewer  (as  shown 
in  Figures  3  to  4)  for  reviewing  and  interpreting  the  outputs  from  the  upward  flow  analysis. 
Specifically,  the  viewer  enhanced  the  value  of  the  feedback  requirements  by  allowing: 

1 .  the  selection  of  a  specific  pair  of  operators  for  closer  examination; 
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2.  the  selection  of  primary  or  secondary  (or  both)  operators  assigned  to  the  goals; 

3.  the  selection  of  all,  or  important,  or  frequent  feedback  requirements;  and 

4.  the  selection  of  feedback  requirements  being  satisfied  visually  or  verbally. 

The  ability  to  view  the  outputs  from  (combinations  of)  these  meaningful  perspectives  helped  the 
analysts,  the  SMEs,  and  the  clients  gain  a  deeper  understanding  of  the  system  and  of  the  solutions 
that  could  be  considered.  Future  applications  of  HGA  would  benefit  from  improved  tools  to 
support  all  phases  of  the  HGA  process,  including  goal  identification  and  assignment,  and  the 
stability  analysis. 
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5  Conclusion 


HGA  was  found  to  be  applicable  to  naval  command  and  control,  since  analysts  and  domain 
experts  were  able  to  identify  a  hierarchy  of  goals  for  the  overall  system  and  to  assign  goals  to 
operators  within  the  system.  These  results  were  validated  by  two  teams  of  SMEs.  Since  the 
Halifax  Class  Frigate  operations  room  included  more  than  1 1  human  operators  and  is  utilized  in 
actual  operations,  this  application  also  demonstrated  that  HGA  could  be  used  to  analyze  a  multi¬ 
operator  system  of  industrial  scale  and  maturity. 

HGA  was  found  to  be  especially  useful  because  of  its  flexibility.  While  it  was  used  to  model  an 
existing  system,  it  provided  a  basis  for  developing  and  for  evaluating  alternate  system 
designs  by  producing  control  requirements  and  feedback  requirements  that  could  be  prioritized 
using  some  reasonable  and  intuitive  metrics.  HGA  did  not  rely  on  pre-defmed  roles  for  the 
operators  as  the  starting  point  for  analysis.  The  resulting  goal  hierarchy  could  be  re-used  by 
analysts/designers  who  might  wish  to  assign  completely  different  operators  to  the  goals,  in  order 
to  consider  system-wide  redesigns  involving  the  addition,  removal  and/or  re-organization  of 
operators.  Requirements  produced  by  the  HGA  were  relevant  to  the  design  of  physical 
workspaces,  of  command  and  control  systems  including  individual  and  shared  displays,  and  of 
visual  and  verbal  modes  of  communication. 

The  results  and  recommendations  from  the  current  application  of  HGA  are  being  used  to  develop 
and  evaluate  prototypes  of  new  designs  for  the  operations  room  (including  three-dimensional 
virtual  reality  prototypes).  The  purpose  of  these  prototypes  is  to  support  the  definition  of 
specifications  for  the  design  of  the  operations  room.  Therefore,  this  application  of  HGA 
is  having  significant  input  into  shaping  the  design  of  an  upgraded  operations  room  to  support 
future  operations. 
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List  of  symbols/abbreviations/acronyms/initialisms 


ARRO 

Air  Raid  Report  Operator 

ASPO 

Anti-Submarine  Plotting  Operator 

ASWC 

Assistant  Sensor  Weapon  Controller 

CO 

Commanding  Officer 

EWS 

Electronic  Warfare  Supervisor 

GDTA 

Goal-Directed  Task  Analysis 

HGA 

Elierarchical  Goal  Analysis 

HTA 

Elierarchical  Task  Analysis 

IMD 

Information  Management  Director 

OOW 

Officer  Of  the  Watch 

ORO 

Operations  Room  Officer 

ORS 

Operations  Room  Supervisor 

RMP 

Recognized  Maritime  Picture 

SAC 

Shipbome  Aircraft  Controller 

SME 

Subject  Matter  Expert 

SWC 

Sensor  Weapon  Controller 

TS 

Track  Supervisor 

UAV 

Uninhabited  Aerial  Vehicle 
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